This invention relates to a method and an apparatus of image processing for graininess or noise suppression and sharpness enhancement of digital images, or specifically to a method and an apparatus by which noise such as graininess in digital images can be suppressed while enhancing their sharpness.
In digital imaging technology which records or captured images on silver halide photographs or the like with an image input scanner and which outputs digital images with an image output printer, considerable deterioration occurs in the sharpness of the output images due to the scanner and the printer. As a corrective procedure, sharpness enhancement is conventionally performed by means of a Laplacian filter or an unsharp masking (USM). However, sharpening the image causes the side effect of enhancing image noises including graininess in the image and electrical noise in the scanner or the like, thereby deteriorating the noises including graininess. The resultant image gives a visually unpleasing impression. Hence, images can be subjected to only moderate sharpness enhancement within a range where the visually unpleasing impression is not given.
Especially, where a silver halide color photographic material for use in photographing includes undeveloped silver halide grains and developed silver grains in addition to color-produced cyan, magenta and yellow dyes, for example in the case of a silver halide color photographic material as mentioned in Examples of the commonly assigned EP-A-800114, when the aforementioned sharpness enhancement in the prior art or arithmetic operations for digital image processing such as color correction and gradation correction is performed, graininess is enhanced due to the undeveloped silver halide grains and developed silver grains included in the image, which makes the image quality unpreferred.
Several methods have been proposed to process digital images such as to remove noisy graininess and enhance their sharpness. Removal of graininess involves an averaging or blurring technique. Then, the blurred grainy pattern will not be pleasing to the eye, or fine structures of the image that are not to be removed will be erased together with the graininess, or artifacts that cause unnatural and strange impressions will occur. For these and other reasons, the conventional techniques for removing graininess are not suitable for application to aesthetic images such as photographs.
For example, in the above-described prior art method of processing images to suppress their graininess while enhancing their sharpness, unsharp masking is employed to enhance the sharpness whereas blurring or smoothing is effected to suppress the graininess, such that a graininess (noise) signal and a contour signal are separated from the original image by signal level and the contour signal is subjected to sharpness enhancement whereas the smoothed region is suppressed in graininess and the smaller signal is regarded as representing graininess and processed accordingly; as a result, signals representing the fine details of the image which are close to signal levels representing graininess, namely, image signals representing the texture of cloths, lawn, the hair on the head and the like, will be suppressed along with the graininess, yielding visually unpleasing images that contain artifacts from image processing. In the conventional image processing technology where blurring or averaging is used as the method of suppressing graininess, a blurred grainy pattern seems to be reduced; however, the blurred grainy pattern spreads and will be recognized as a visually unpleasing pattern, which stands out markedly in someone""s face or skin as in portraits or in solid subjects such as walls or sky.
The present invention has been accomplished under these circumstances. Images such as those in photographs, in printed documents, on television""s screens, in digital still photographs and from various kinds of copiers, especially photographic images including the above-described undeveloped silver halide grains and developed silver grains suffer from the camera-induced blur, the deterioration in noise and sharpness which is inherent in the image as exemplified by graininess and blur in photographic materials or the above-mentioned defects encountered when digitizing the original image with an image input device such as a scanner, that is, graininess is enhanced to produce a visually unpleasing impression; low-contrast image signals are mistaken for graininess and suppressed or entirely eliminated; and the boundary between a region from which graininess has been removed and a region where sharpness enhancement has been done becomes discontinuous to produce unnatural artifacts in the output image. The purpose of the present invention is to provide a method and an apparatus for processing digital images to suppress noise and enhance sharpness, by which graininess can be suppressed and image sharpness enhanced without causing any of the defects associated with the prior art.
In order to achieve the above purpose, according to the present invention, there is provided a method of processing a digital image for noise suppression and sharpness enhancement, comprising the steps of:
performing first a sharpness enhancing process on original image data of an original image to create sharpness enhanced image data in which an image and a noise included therein are both sharpened;
performing a smoothing process on the original image data to create smoothed image data;
subtracting the smoothed image data from the sharpness enhanced image data to create image data containing subject image edges and the noise of which the sharpness enhancement is achieved;
performing an edge detection from the original image data to determine weighting data for an edge region and weighting data for a noise region used to discriminate the edge region of a subject and the noise region;
multiplying the image data containing the subject image edges and the noise by the weighting data for the noise region to determine noise data of each color in the noise region;
subsequently determining a black-and-white noise component and a color noise component from the noise data of the each color;
multiplying the thus obtained black-and-white noise component and color noise component by their suppressing coefficients to determine a black-and-white noise suppressing component and a color noise suppressing component; and
selectively removing the black-and-white noise suppressing component and the color noise suppressing component from the sharpness enhanced image data, thereby creating a processed image in which the noise suppression is achieved while retaining the sharpness enhancement in the edge region of the image.
In the image processing method according to a first embodiment of the invention, it is preferred that the black-and-white noise component and the color noise component are respectively determined as a component which exists equally in the each color and as a component which exists independently in the each color, based on a color correlation component obtained by calculating a color correlation of the noise data of the each color in the noise region.
The weighting data for the noise region is preferably determined from the weighting data for the edge region.
The present invention also provides an apparatus for processing a digital image for noise suppression and sharpness enhancement, comprising:
a sharpness enhancing unit which performs a sharpness enhancing process on original image data of an original image to create sharpness enhanced image data in which an image and a noise included therein are both sharpened;
a smoothing unit which performs a smoothing process on the original image data to create smoothed image data;
an edge/noise component extracting unit which subtracts the smoothed image data from the sharpness enhanced image data to create image data containing subject image edges and the noise of which the sharpness enhancement is achieved;
an edge detecting unit which performs an edge detection from the original image data to determine weighting data for an edge region used to discriminate the edge region of a subject and a noise region;
a weighting coefficient calculating unit for the noise region in which weighting data for the noise region is determined from the weighting data for the edge region;
a noise component discriminating unit in which the image data containing the subject image edges and the noise is multiplied by the weighting data for the noise region to determine noise data of each color in the noise region, from which a black-and-white noise component and a color noise component are determined, after which the thus obtained black-and-white noise component and color noise component are multiplied by their suppressing coefficients to determine a black-and-white noise suppressing component and a color noise suppressing component; and
an output image calculating unit which selectively removes the black-and-white noise suppressing component and the color noise suppressing component from the sharpness enhanced image data, thereby creating a processed image in which the noise suppression is achieved while retaining the sharpness enhancement in the edge region of the image.
In the image processing apparatus according to the first embodiment of the invention, the noise component discriminating unit determines preferably the black-and-white noise component which exists equally in the each color and the color noise component based on a color correlation component obtained by calculating a color correlation of the noise data of the each color in the noise region.
In the image processing method and apparatus according to the first embodiment of the invention, preferably, the noise includes graininess; the noise region, the black-and-white noise component and the color noise component are a grainy region, a black-and-white grain component and a dye grain component, respectively; the original image is an image recorded on a silver halide color photographic material; and the black-and-white grain component includes at least one of undeveloped silver halide grains and developed silver grains.
It is also preferred that the original image data is image data captured with an image pickup device such as a scanner, an imaging device or imaging elements from an image recorded on a silver halide color photographic material, and that the black-and-white noise component includes not only a black-and-white grain component formed of at least one of undeveloped silver halide grains and developed silver grains, but also a fixed pattern noise of the image pickup device (the scanner, the imaging device or the imaging elements).
It is further preferred that the original image is an image recorded with an image pickup device such as a scanner, an imaging device or imaging elements, and that the black-and-white noise component is a fixed pattern noise of the image pickup device (the scanner, the imaging device or the imaging elements), which is strong in the color correlation, whereas the color noise component is a noise which is weak in the color correlation.
In the image processing method according to a second embodiment of the invention, it is preferred that the noise includes graininess, and the noise region, the noise data, the black-and-white noise component, the color noise component, the black-and-white noise suppressing component and the color noise suppressing component are a grainy region, grain data, a black-and-white grain component, a dye grain component, a black-and-white grain suppressing component and a dye grain suppressing component, respectively, and that a local grain coefficient representing a spatial magnitude and a variation magnitude of density variations due to the graininess is determined and used to discriminate and separate the black-and-white grain component and the dye grain component from the grain data of the each color in the grainy region.
That is, the second embodiment of the invention provides a method of processing a digital image for noise suppression and sharpness enhancement, comprising the steps of:
performing first a sharpness enhancing process on original image data of an original image to create sharpness enhanced image data in which the image and a noise or graininess included therein are both sharpened;
performing a smoothing process on the original image data to create smoothed image data;
subtracting the smoothed image data from the sharpness enhanced image data to create image data containing subject image edges and the grain of which the sharpness enhancement is achieved;
performing an edge detection from the original image data to determine weighting data for an edge region and weighting data for a grainy region used to discriminate the edge region of a subject and the grainy region;
multiplying the image data containing the subject image edges and the grain by the weighting data for the grainy region to determine grain data of each color in the grainy region;
subsequently determining a local grain coefficient representing a spatial magnitude and a variation magnitude of density variations due to the graininess to discriminate and separate the black-and-white grain component and the dye grain component from the grain data of the each color;
multiplying the thus obtained black-and-white grain component and dye grain component by their suppressing coefficients to determine a black-and-white grain suppressing component and a dye grain suppressing component; and
selectively removing the black-and-white grain suppressing component and the dye grain suppressing component from the sharpness enhanced image data, thereby creating a processed image in which the graininess suppression is achieved while retaining the sharpness enhancement in the edge region of the image.
In the image processing apparatus according to the second embodiment of the invention, it is preferred that the noise includes graininess, and the noise region, the noise data, the black-and-white noise component, the color noise component, the black-and-white noise suppressing component, the color noise suppressing component, the edge/noise component extracting unit, the weighting coefficient calculating unit for the noise region and the noise component discriminating unit are a grainy region, grain data, a black-and-white grain component, a dye grain component, a black-and-white grain suppressing component, a dye grain suppressing component, an edge/grain component extracting subsection, a weighting coefficient calculating subsection for the grainy region and a processing subsection for discriminating a grain component, respectively; and that the processing subsection for discriminating the grain component determines a local grain coefficient representing a spatial magnitude and a variation magnitude of density variations due to the graininess to discriminate and separate the black-and-white grain component and the dye grain component from the grain data of the each color in the grainy region.
That is, the second embodiment of the invention provides an apparatus for processing a digital image for noise suppression and sharpness enhancement, comprising:
a sharpness enhancing unit which performs a sharpness enhancing process on original image data of an original image to create sharpness enhanced image data in which an image and a noise or graininess included therein are both sharpened;
a smoothing unit which performs a smoothing process on the original image data to create smoothed image data;
an edge/grain component extracting unit which subtracts the smoothed image data from the sharpness enhanced image data to create image data containing subject image edges and the grain of which the sharpness enhancement is achieved;
an edge detecting unit which performs an edge detection from the original image data to determine weighting data for an edge region used to discriminate the edge region of a subject and a grainy region;
a weighting coefficient calculating unit for the grainy region in which weighting data for the grainy region is determined from the weighting data for the edge region;
a grain component discriminating unit in which the image data containing the subject image edges and the grain is multiplied by the weighting data for the grainy region to determine grain data of each color in the grainy region, from which a local grain coefficient representing a spatial magnitude and a variation magnitude of density variations due to the graininess is determined to discriminate and separate the black-and-white grain component and the dye grain component, after which the thus obtained black-and-white grain component and dye grain component are multiplied by their suppressing coefficients to determine a black-and-white grain suppressing component and a dye grain suppressing component; and
an output image calculating unit for selectively removing the black-and-white grain suppressing component and the dye grain suppressing component from the sharpness enhanced image data, thereby creating a processed image in which the graininess suppression is achieved while retaining the sharpness enhancement in the edge region of the image.
In the image processing method and apparatus according to the second embodiment of the invention, it is preferred that the original image is an image recorded on a silver halide color photographic material and that the black-and-white grain component includes at least one of undeveloped silver halide grains and developed silver grains.
It is also preferred that the original image data is image data captured with an image pickup device such as a scanner, an imaging device or imaging elements from an image recorded on a silver halide color photographic material, and that the black-and-white grain component includes not only a grain component formed of at least one of undeveloped silver halide grains and developed silver grains, but also at least one of a random noise of the each color, a fixed pattern noise of the image pickup device (the scanner, the imaging device or the imaging elements) and moire due to aliasing.
Further, in the image processing methods and apparatuses in the first and second embodiments of the invention, it is preferred that the edge detection is performed by a local variance, and that the sharpness enhancing process and the smoothing process are Gaussian unsharp masking and Gaussian masking, respectively. However, these are not of course the sole cases of the invention, and other techniques may be used.
The sharpness enhancement is preferably applied in a necessary and sufficiently intense manner, although graininess is considerably marked without being suppressed.
In the present invention, the sharpness enhancing process is first performed on a color original image to sharpen the image and graininess or noise included therein, after which the edge region and the flat region of the image are divided. The flat region is then regarded as the noise region or grainy region to detect noise or grain signals.
Then, in the first embodiment of the invention, for example, color correlation of the noise or grain signals of red (R), green (G) and blue (B) is calculated, and the thus obtained color correlation component is regarded as the black-and-white noise (or black-and-white grain) which is equally included in the three colors, if the color correlation is strong, and as the color noise (or dye grain) which is independently included therein, if the color correlation is weak. Thus, the two noises (or grains) are discriminated. The thus discriminated black-and-white noise (or black-and-white grain) can be selectively removed from the noise (or grainy) region of the sharpness enhanced image signals of the three colors, thereby removing for example the black-and-white grain component formed of undeveloped silver halide grains and developed silver grains in a silver halide color photographic material, and other black-and-white noise component such as a fixed pattern noise in a scanner or imaging device. On the other hand, the remaining color noise (or dye grain) is subjected to the noise (or graininess) suppression for each color.
In the second embodiment of the invention, for example, characteristic quantities of the signals produced by a morphological difference between the black-and-white grain and the dye grain are calculated from the grain signals of R, G and B. Thus, the two grains are discriminated. The thus discriminated black-and-white grain can be selectively removed from the grainy region of the sharpness enhanced image signals of the three colors, thereby removing the black-and-white grain component formed of undeveloped silver halide grains and developed silver grains in a silver halide color photographic material. On the other hand, the dye grain is subjected to the graininess suppression for each color of R, G and B.
According to the method for noise (or graininess) suppression as applied to the image processing method and apparatus of the invention, the noise (or grain) component is also finely subdivided by sharpness enhancement together with the subject component of the image. The noise (or graininess) is suppressed by a method of subtracting the noise (or grain) component from the sharpness enhanced image. Therefore, the noise (or graininess) achieved by this technique is spatially finer and lower in density contrast than it initially was. Since the noise (or graininess) is refined spatially, it can be as fine as is achieved when fine-grain emulsions are used to process silver halide color photographic materials.